Mathematical Optimization: all about derivatives part 1

In this brief post we talk about why derivatives are so important in machine learning / deep learning, and begin our discussion of them by using pictures and interactive widgets (and very little math).

The widget below shows a function along with the tangent line defined by its derivative at a point. The slider animates the point at which the derivative is taken and illustrated.



Again as you slide from left to right you can see how the line defined by the derivative at each point stays tangent to the curve, hugs the function's shape everywhere, and generally matches the function near the point.

And as mentioned previously - this notion holds for functions of any dimension. The only difference is that our tangent line becomes a hyperplane. For example the simple sinusoid in 3 dimensions given by

\begin{equation} g(w_0, w_1) = \text{sin}(w_0) \end{equation}

The next Python cell produces a static plot illustrating this function (shown in gray) along with the tangent hyperplane (in green) defined by its derivative at a specific point (in red). Again the function, as well as the given point, can be adjusted at will in the Jupyter notebook version of this post.

In [4]:
# define the function to plot, as well as a point at which to draw tangent hyperplane
g = lambda w: np.sin(w[0])
w_val = [-1.5,1]

# load in function to examine
taylor_viz = calclib.taylor3d_viz.visualizer(g = g)

# start examination
taylor_viz.draw_it(w_val = w_val,first_order = True,view = [20,110]);